Variational Bayes via Propositionalization

In this paper, we propose a unified approach to VB (variational Bayes) [1] in symbolicstatistical modeling via propositionalization1. By propositionalization we mean, broadly, expressing and computing probabilistic models such as BNs (Bayesian networks) [2] and PCFGs (probabilistic context free grammars) [3] in terms of propositional logic that considers propositional variables as binary random variables. Our proposal is motivated by three observations. The first one is that propositionalization, or more precisely PPC (propositionalized probability computation), i.e. probability computation formulated in a propositional setting, has turned out to be general and efficient when variable values are sparsely interdependent. Examples include (discrete) BNs, PCFGs and more generally PRISM [4, 5] which is a probabilistic logic programming language we have been developing that computes probabilities using graphically represented AND/OR boolean formulas. Efficacy of PPC is already proved by the Inside-Outside algorithm in the case of PCFGs and by recent PPC approaches to BNs such as the one by Chavira et al. that exploits 0 probability and CSI (context specific independence) [6]. Mateescu et al. introduced AND/OR search tress which is a propositional representation of bucket trees and revealed that PPC is a general computation machanism for BNs [7]. Second of all, while VB has been around for sometime as a powerful tool for Bayesian modeling [1], it’s use is restricted to somewhat simple models such as BNs and HMMs (hidden Markov models) [8] though its usefulness is established through a variety of applications from model selection to prediction. On the other hand it is already proved that VB can be extended to PCFGs and efficiently implementable using dynamic programming [9]. Note that PCFGs are just one class of PPC and much more general PPC is already realized in PRISM. Accordingly if VB is combined with PRISM’s PPC, we will obtain VB for general probabilistic models, far wider than BNs and PCFGs. The last observation is that currently deriving and implementing a VB algorithm is an error-prone time-consuming process. In addition ensuring its correctness beyond PCFGs seems a non-trivial task. However once VB becomes available in PRISM, it will save considerable time and effort. That is, we do not have to derive a new VB algorithm from scratch nor have to implement it. All we have to do is just to write a probabilistic